To PAC and Beyond

The greatest advantage of learning algorithms is their ability to adjust themselves to the required task. While “traditional” algorithms are tailored to perform a single task, learning algorithms acquire the capability to perform it. Learning techniques have been applied successfully in various domains ranging from optical character recognition and information retrieval to medical diagnostics and fraud detection. Although there are different frameworks for learning, the most common one is the model of learning from examples. In this framework, the learner is presented with a set of observations and the required action (or label) for each of these observations. The learner needs to learn the map between observations and labels. For example, in an optical character recognition task, an observation can be a bit-map representation of a written character and the label is the name of this character. This framework is intuitive to the teacher (typically human) and learner (the machine) However, the length of the learning process makes it unfeasible in many cases. The teacher is required to present a large set of example before an acceptable learning level is reached. Consider for example the task of teaching a machine to perform a medical diagnosis. An example in this case is a full profile of a patient, and the label is a medical diagnosis of his or her medical condition. Supplying these data is a labor-intensive task, which requires experts. Active Learning tries to shorten the learning process by allowing the learner some control over the learning process. This control enables the learner to direct the teacher to the areas in which the learner is less confident. One possible method of applying active learning is via Membership Queries. In this model the learner can construct observations and ask the teacher to label them. This model has been proved successful in solving theoretical problems such as constant depth circuit learning [75], decision tree learning [65] and others. However, membership queries work poorly when the teacher is human [67]. This deficiency is due to the fact that many of the queries directed by the learner are not clear to the human teacher since they lack the consistency